In general, most operating systems include a kernel configured for providing the core services, and a shell or applications that use the kernel services and provide a user interface. In addition, most operating systems have two modes of operation: a privileged level where the kernel resides, and an unprivileged level where the application and system processes execute. A ring transition mechanism is used to move from one mode to the other.
A system call is a mechanism with which application and system processes access the services provided by the operating system. A system call typically involves a ring transition to the kernel where the required function is then performed. Traditionally this is achieved by using a trap (interrupt) mechanism that transitions into the kernel to execute the required function. Intel introduced the SYSENTER/SYSEXIT based fast system call mechanism (FSCM) for its x86 processor architectures to efficiently handle ring-transition oriented system calls. Similarly, Advanced Micro Devices has a SYSCALL/SYSRET based FSCM for its x64 processor architectures to efficiently handle ring-transition oriented system calls.
For instance, on Windows XP systems, every thread in the system contains an embedded pointer to a table of functions on x86 systems (or offsets to functions on x64 systems) that an FSCM utilizes to dispatch operating system calls from user mode applications. Other multithreading operating systems have similar functionality and mechanisms.
The FSCM essentially performs the same function on most computing systems, although it typically is implemented in a slightly different manner depending on the processor of the system (e.g., x64 processors by Advanced Micro Devices implement FSCM slightly different from how x86 processors by Intel FSCM). When a SYSENTER on x84 (or SYSCALL on x64) instruction is emitted, the processor's instruction pointer is set to a value stored in a model specific register (MSR). This register is populated very early in the operating system's startup process. The routine pointed to by the FSCM MSR is a highly optimized operating system (OS) specific routine that handles among other tasks, dispatching the call using the correct system call table for the thread currently being executed.
What is needed are techniques that exploit such system call mechanisms to effect robust security applications on targeted processes/threads. Such techniques should operate without impacting performance of any other running processes/threads.